Unfortunately this won't work for editable installs. The py files and the compiled extension modules are not side-by-side for the mode of editable installation we are using. The right way to make this work is to use importlib.resources to get all of the file paths without loading the module, but that will require some work upstream in scikit-build-core to support that.

How are you currently performing an editable install? The PTX file locations aren't relative to any extension module, they should just be relative to the py files themselves.

I'm doing an editable installation with pip install -e.... Try it out and see where files end up. I'm happy to show it to you on my machine if you'd like or if you have trouble reproducing the behavior I'm mentioning.

The problem is that the PTX files are generated at build time and installed via CMake. When scikit-build-core does an editable install, anything that gets installed by CMake is placed into a directory in site-packages, while any pure Python files are read directly from the source directory (as would be expected with an editable install). Observe:

(rapids) coder _ ~/cudf $ pwd
/home/coder/cudf
(rapids) coder _ ~/cudf $ find . -name "*.ptx"
./cpp/build/release/CMakeFiles/3.28.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx
./cpp/build/conda/cuda-12.2/release/CMakeFiles/3.29.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx
./cpp/build/conda/cuda-12.2/release/CMakeFiles/3.29.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx
./python/cudf/build/cp310-cp310-linux_x86_64/CMakeFiles/3.29.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx
./python/cudf/build/cp310-cp310-linux_x86_64/CMakeFiles/3.29.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx
./python/cudf/build/cp310-cp310-linux_x86_64/udf_cpp/CMakeFiles/shim_60.dir/shim.ptx
./python/cudf/build/cp310-cp310-linux_x86_64/udf_cpp/CMakeFiles/shim_80.dir/shim.ptx
./python/cudf/build/cp310-cp310-linux_x86_64/udf_cpp/CMakeFiles/shim_75.dir/shim.ptx
./python/cudf/build/cp310-cp310-linux_x86_64/udf_cpp/CMakeFiles/shim_70.dir/shim.ptx
./python/cudf/build/cp310-cp310-linux_x86_64/udf_cpp/CMakeFiles/shim_90.dir/shim.ptx
./python/cudf/build/cp310-cp310-linux_x86_64/udf_cpp/CMakeFiles/shim_86.dir/shim.ptx
(rapids) coder _ ~/cudf $ find $(python -c "import site; print(site.getsitepackages()[0])")/cudf -name "*.ptx"
/home/coder/.conda/envs/rapids/lib/python3.10/site-packages/cudf/core/udf/shim_80.ptx
/home/coder/.conda/envs/rapids/lib/python3.10/site-packages/cudf/core/udf/shim_86.ptx
/home/coder/.conda/envs/rapids/lib/python3.10/site-packages/cudf/core/udf/shim_60.ptx
/home/coder/.conda/envs/rapids/lib/python3.10/site-packages/cudf/core/udf/shim_70.ptx
/home/coder/.conda/envs/rapids/lib/python3.10/site-packages/cudf/core/udf/shim_75.ptx
/home/coder/.conda/envs/rapids/lib/python3.10/site-packages/cudf/core/udf/shim_90.ptx

I see - if the cython extension modules are the only thing we can guarantee the location of in an editable install, I'd like to propose we ship some kind of empty cython module along with _lib (_path perhaps?) for this purpose - one that does not import the full _lib namespace - what do you think @vyasr ? If that sounds like a reasonable path forward, I can adapt this PR for that goal, else, I'll create an issue where we can figure things out in more detail and come back to this later.

I guess it depends, what exactly is the problem that you're trying to solve? Is there a way to delay the set up of the PTX files until after you know cudf has been imported? That way you wouldn't have any need to import cudf prematurely. We could add a stub library, but I feel like that's going to be more trouble than it's worth. If we did move forward with that, you'd want to put a pyx file in core/udf with a function get_ptx_files or something so that it can always be colocated with the PTX files.

I've asked in scikit-build about how best to do this since I do think that's where work needs to be done to properly enable support.

We're only reading from a PTX file because the version of CUDA used to compile it is needed to determine if MVC is necessary. This is the case where our package is compiled with CUDA V.X and the user has driver version V.Y, where X > Y. Unfortunately I think that means that we can't import cudf first, since later on in the cuDF import process we import numba.cuda after which we can't patch the linker. So, this process must happen before the rest of cuDF initializes. I'm of the opinion that we should aspire to fix this, mostly because I think in the current state it might be confusing to users exactly what piece of the initialization process is going wrong.

While we're waiting to hear back re:scikit-build, maybe a try-except could do the trick as a patch too? I'm not sure that I see a way for the empty lib approach to work because just doing import cudf._lib.empty_lib will trigger a full import of everything in cudf AFAIK. We import too many things in __init__.py files.

I did some more investigating on the skbuild side and followed up with scikit-build/scikit-build-core#807 and scikit-build/scikit-build-core#808.

I'm going to close this since we should really be aiming to fix this upstream in scikit-build-core.